Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[UI Tests] - Add ensure_devices_found parameter to fastlane scan #14767

Open
wants to merge 14 commits into
base: trunk
Choose a base branch
from

Conversation

jostnes
Copy link
Contributor

@jostnes jostnes commented Dec 26, 2024

Description

The UI test suite is still flaky, with the test suite reliability over the last 7 days at only 62.5%. I checked the last few failing builds (merges to trunk) and found that:

  1. 3x because of the error: WooCommerceUITests-Runner (2458) encountered an error (The test runner failed to initialize for UI testing. (Underlying Error: Timed out waiting for AX loaded notification))
  2. 2x because of timeout/slowness in CI
  3. 1x because of the error: WooCommerceUITests-Runner (2566) encountered an error (Early unexpected exit, operation never finished bootstrapping - no restart will be attempted. (Underlying Error: Crash: WooCommerceUITests-Runner (2566) <external symbol>))

The error Underlying Error: Timed out waiting for AX loaded notification looks to be related to the simulator not being ready before running the test, previously we added a check to make sure that the simulator is ready before the test starts. But missed the part that we're using fastlane's scan to run the test.

Checking the fastlane doc, I found that there's a parameter that we could use to ensure that all simulators (since we are running with concurrency = 2) are ready before the test starts. This PR adds ensure_devices_found: true and replaced reset_simulator: true (which can cause tests to start slower) to force_quit_simulator: true.

I also updated the two timeout values of the two slowness screens from the default value of 20 seconds to 35 seconds. After running the test suite at least 5x, with the new parameters, I didn't encounter any failure with the error Timed out waiting for AX loaded notification. There are still timeouts happening to other screens due to CI slowness, but we can look at that again after the MacOS agent update and leave that out of the scope of this PR.

Steps to reproduce

UI test suite is failing intermittently in CI.

Testing information

CI should be 🟢 . I also ran the test suite multiple times in CI and checked that this error didn't occur (testing builds):

Testing failed:
--
  | WooCommerceUITests-Runner (2458) encountered an error (The test runner failed to initialize for UI testing. (Underlying Error: Timed out waiting for AX loaded notification))

Screenshots


  • I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

Reviewer (or Author, in the case of optional code reviews):

Please make sure these conditions are met before approving the PR, or request changes if the PR needs improvement:

  • The PR is small and has a clear, single focus, or a valid explanation is provided in the description. If needed, please request to split it into smaller PRs.
  • Ensure Adequate Unit Test Coverage: The changes are reasonably covered by unit tests or an explanation is provided in the PR description.
  • Manual Testing: The author listed all the tests they ran, including smoke tests when needed (e.g., for refactorings). The reviewer confirmed that the PR works as expected on all devices (phone/tablet) and no regressions are added.

@wpmobilebot
Copy link
Collaborator

wpmobilebot commented Dec 26, 2024

WooCommerce iOS📲 You can test the changes from this Pull Request in WooCommerce iOS by scanning the QR code below to install the corresponding build.

App NameWooCommerce iOS WooCommerce iOS
Build Numberpr14767-dab3ea7
Version21.3
Bundle IDcom.automattic.alpha.woocommerce
Commitdab3ea7
App Center BuildWooCommerce - Prototype Builds #12298
Automatticians: You can use our internal self-serve MC tool to give yourself access to App Center if needed.

@jostnes jostnes changed the title Testing in CI [UI Tests] - Add ensure_devices_found parameter to fastlane scan Dec 27, 2024
@jostnes jostnes added type: task An internally driven task. category: ui tests Related to UI testing. labels Dec 27, 2024
@jostnes jostnes added this to the 21.4 milestone Dec 27, 2024
@jostnes jostnes marked this pull request as ready for review December 27, 2024 04:46
@jostnes jostnes requested a review from a team as a code owner December 27, 2024 04:46
@jostnes jostnes requested a review from a team December 27, 2024 04:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: ui tests Related to UI testing. type: task An internally driven task.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants